During the last 10 years, the Internet has become an all encompassing means for connecting computers globally. The ever increasing use of the Internet has resulted in more and more services being offered, and the increasing number of computers connected to the network has caused a veritable explosion in the amount of data transferred.
Another factor in the use of the public Internet is the presence of a great variety of malicious programs like viruses and worms or the like, which spread across the Internet. Detection and blocking of these programs is extremely important, not only for the private user, but even more for corporations, which rely almost entirely on a fully functional network, both internally for their users and externally with customers and partners. An infection of a corporate network with malicious software can be very costly and can render a corporate data network unusable for days and invaluable data may be lost. This can have a major financial impact for the corporation.
Also the providers of data networks like Internet Service Providers (ISPs) face the task of protecting their networks from infection. ISPs provide Internet connections to a large number of customers and can carry a significant amount of Internet traffic on their backbone. Backbone links with capacity of 10 Gbps or more are now possible. The ISPs must employ means for protecting their infrastructure from interruption, thus causing downtime for the customers. They can also in some cases be contractually obliged to ensure that no malicious software penetrate from their network into their customer corporate network.
For corporate users and ISPs alike, it should be possible to apply protection to parts of the infrastructure, wherein data traffic from a large number of users flow.
Historically these kinds of protection means have been implemented on the lower levels of the OSI model. Protection was provided by blocking or allowing specific ranges of IP addresses or TCP/UDP ports, i.e. layer 3 and layer 4 of the OSI model. However, this approach has turned out to be much too crude to provide sufficient protection.
Newer methods involve methods for inspecting the data traffic on higher levels of the OSI model. This means that the employed devices must be able to recognise specific malicious data patterns all the way up layer 7 in the OSI model, which also means that the devices must be able to extract streams belonging to the same user and application from the data traffic. Furthermore, they must be able to inspect each stream in parallel with the streams from all other users and applications.
Pattern matching is currently implemented using software centric solutions which typically have large flexibility but suffer from low throughput.
Other pattern match methods typically de-compress the data before the actual search is carried out. This has the undesired effect that the necessary bandwidth may have to be many times the bandwidth of the incoming data. Therefore, searching in such de-compressed data is difficult or impossible to perform at line speed.
Dedicated hardware solutions have also been presented but typically they do not allow the use of advanced search combinations and regular expression matching. US 2005/0154802 A1 describes a parallel pattern detection engine (PPDE) using multiple underlying processor units (PU) to search for various patterns. The PUs can be utilised in single mode or cascade mode. Cascade mode supports longer patterns to be matched or more patterns to be matched.
Specifically targeting string matching applications, the invention focuses on processing data at high speed (>10 Gbps). Internet contents are usually compressed before trans-mission in order to obtain low bandwidth and low latency. This approach is possible due to the increased processing capabilities found in standard PCs. The high processing power can be used to decompress the Internet contents.
In general, decompression is difficult to perform at wire-speed due to the additional data-amounts generated, which means that a decompressed 10 Gbps link may contain e.g. 30-40 Gigabit data per second, which must be stored and processed during the packet inspection process. Depending on the packet inspection method employed, this is either not possible at all, or results in high cost of the inspection device due to the large amount of storage and processing power.